Information processing apparatus, information processing method, and computer program

ABSTRACT

There is provided an information processing apparatus including an operation detection unit configured to detect an orientation of a user&#39;s face and operations performed by the user, and an area selection unit configured to, when the operation detection unit detects that the user has performed a first operation, select an area on a screen based on the orientation of the user&#39;s face during the first operation.

CROSS-REFERENCE TO PRIOR APPLICATION

This application is a continuation of U.S. patent application Ser. No.13/758,244 (filed on Feb. 4, 2013), which claims priority to JapanesePatent Application No. 2012-076211 (filed on Mar. 29, 2012), which areall hereby incorporated by reference in their entirety.

BACKGROUND

The present disclosure relates to an information processing apparatus,an information processing method, and a computer program.

Technology exists for operating a television, a personal computer andthe like based on user gestures without the use of a keyboard, a mouse,or a remote controller (refer to JP 2008-146243A, JP 2007-310914A, andJP 2011-513847T, for example). If such a technology is used, atelevision, a personal computer and the like can be remotely operatedwithout the use of an operation unit, such as a keyboard, a mouse, or aremote controller.

For example, the technology described in JP 2008-146243A executes mostfunctions using a few gestures to stop unintended functions. Further,the technology described in JP 2007-310914A executes a remote operationby detecting movement of facial parts. In addition, the technologydescribed in JP 2011-513847T provides selection of underlying iconfunctionality through straightforward, intuitive, and natural motions ofthe user's body.

SUMMARY

Thus, although technology exists for remote operation of a television, apersonal computer and the like based on user gestures, when performing aremote operation by detecting only the orientation of a face or movementof facial parts, the large movements of the cursor displayed on a largescreen of 50-inches or more, for example, make it different to preciselyspecify a position. Further, when performing a remote operation bydetecting only the user's hand movements, although precise movements ofthe cursor displayed on the screen may be possible, large movements ofthe cursor displayed on the screen are difficult.

Therefore, according to an embodiment of the present disclosure,provided are a new and improved information processing apparatus,information processing method, and computer program, which are capableof facilitating a remote operation performed based on detection of aspecific gesture when a user performs a gesture-based remote operation.

According to an embodiment of the present disclosure, there is providedan information processing apparatus including an operation detectionunit configured to detect an orientation of a user's face and operationsperformed by the user, and an area selection unit configured to, whenthe operation detection unit detects that the user has performed a firstoperation, select an area on a screen based on the orientation of theuser's face during the first operation.

According to such a configuration, an operation detection unit detectsthe orientation of a user's face and operations performed by the user.When the operation detection unit detects that the user has performed afirst operation, an area selection unit selects an area on the screenbased on the orientation of the user's face during the first operation.Consequently, the information processing apparatus can facilitate aremote operation that is performed based on detection of a specificgesture when a user performs a gesture-based remote operation.

Further, according to an embodiment of the present disclosure, there isprovided an information processing method including detecting anorientation of a user's face and operations performed by the user, andwhen performance of a first operation by the user is detected, selectingan area on a screen based on the orientation of the user's face duringthe first operation.

Further, according to an embodiment of the present disclosure, there isprovided a computer program configured to make a computer detect anorientation of a user's face and operations performed by the user, andwhen performance of a first operation by the user is detected, select anarea on a screen based on the orientation of the user's face during thefirst operation.

Thus, according to the embodiments of the present disclosure describedabove, a new and improved information processing apparatus, informationprocessing method, and computer program can be provided that are capableof facilitating a remote operation performed based on detection of aspecific gesture when a user performs a gesture-based remote operation.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an explanatory diagram illustrating an appearance example of adisplay apparatus 100 according to an embodiment of the presentdisclosure;

FIG. 2 is an explanatory diagram illustrating a function configurationexample of a display apparatus 100 according to an embodiment of thepresent disclosure;

FIG. 3 is an explanatory diagram illustrating a function configurationexample of an operation detection unit 140 included in a displayapparatus 100 according to an embodiment of the present disclosure;

FIG. 4 is an explanatory diagram illustrating conversion processingperformed by a face detection unit 142;

FIG. 5 is an explanatory diagram illustrating conversion processingperformed by a face detection unit 142;

FIG. 6 is an explanatory diagram illustrating conversion processingperformed by a face detection unit 142;

FIG. 7 is an explanatory diagram illustrating calculation of a positionon a display area of a display unit 110 indicated by a face detected bya face detection unit 142;

FIG. 8 is an explanatory diagram illustrating conversion processingperformed by a face detection unit 142;

FIG. 9A is an explanatory diagram illustrating a control example of adisplay apparatus 100 according to an embodiment of the presentdisclosure;

FIG. 9B is an explanatory diagram illustrating a control example of adisplay apparatus 100 according to an embodiment of the presentdisclosure;

FIG. 9C is an explanatory diagram illustrating a control example of adisplay apparatus 100 according to an embodiment of the presentdisclosure;

FIG. 10 is a flow diagram illustrating operation of a display apparatus100 according to an embodiment of the present disclosure;

FIG. 11 is an explanatory diagram illustrating an example of a screendisplayed on a display unit 110 in a display apparatus 100;

FIG. 12 is an explanatory diagram illustrating an example of a screendisplayed on a display unit 110 in a display apparatus 100;

FIG. 13A is an explanatory diagram illustrating an example of a screendisplayed on a display unit 110 in a display apparatus 100;

FIG. 13B is an explanatory diagram illustrating an example of a screendisplayed on a display unit 110 in a display apparatus 100; and

FIG. 14 is a block diagram illustrating a hardware configuration of adisplay apparatus 100 according to an embodiment of the presentdisclosure.

DETAILED DESCRIPTION OF THE EMBODIMENT(S)

Hereinafter, preferred embodiments of the present disclosure will bedescribed in detail with reference to the appended drawings. Note that,in this specification and the appended drawings, structural elementsthat have substantially the same function and structure are denoted withthe same reference numerals, and repeated explanation of thesestructural elements is omitted.

The description will be made based on the following order.

<1. Embodiment of the Present Disclosure> [Display Apparatus AppearanceExample] [Display Apparatus Function Configuration Example] [DisplayApparatus Operation] <2. Summary> 1. Embodiment of the PresentDisclosure Display Apparatus Appearance Example

First, an appearance example of a display apparatus according to anembodiment of the present disclosure will be described. FIG. 1 is anexplanatory diagram illustrating an appearance example of a displayapparatus 100 according to an embodiment of the present disclosure. Theappearance of the display apparatus 100 according to an embodiment ofthe present disclosure will now be described with reference to FIG. 1.

The display apparatus 100 illustrated in FIG. 1, which is an example ofthe information processing apparatus according to the presentdisclosure, detects user gestures and executes processing based on thosegestures. As illustrated in FIG. 1, the display apparatus 100 accordingto an embodiment of the present disclosure includes a display unit 110and a camera 120.

The display unit 110 displays images based on a supplied signal. Thecamera 120, which is provided in the same direction as the displaysurface of the display unit 110, captures images of objects facing thedisplay surface of the display unit 110. The display apparatus 100illustrated in FIG. 1 is configured to detect user gestures from imagescaptured by the camera 120, and execute processing based on thosegestures.

The display apparatus 100 illustrated in FIG. 1 may be any device, aslong as it is configured to detect user gestures from images captured bythe camera 120, and execute processing based on those gestures. Forexample, this display apparatus 100 may be a television, a monitorconnected to a personal computer or some other information processingapparatus, or a projector.

An appearance example of the display apparatus 100 according to anembodiment of the present disclosure was described above with referenceto FIG. 1. Next, a function configuration example of the displayapparatus 100 according to an embodiment of the present disclosure willbe described.

[Display Apparatus Function Configuration Example]

FIG. 2 is an explanatory diagram illustrating a function configurationexample of the display apparatus 100 according to an embodiment of thepresent disclosure. A function configuration example of the displayapparatus 100 according to an embodiment of the present disclosure willnow be described with reference to FIG. 2.

As illustrated in FIG. 2, the display apparatus 100 according to anembodiment of the present disclosure includes the display unit 110, thecamera 120, an image processing unit 130, an operation detection unit140, an area selection unit 150, and an information display unit 160.

As described above, the display unit 110 displays images based on asupplied signal. In this embodiment according to the present disclosure,the display unit 110 displays information based on a signal suppliedfrom the information display unit 160. Examples of the display unit 110include display devices such as a liquid crystal display panel, anorganic EL display panel, and a plasma display panel. If the displayapparatus 100 is an apparatus like a projector that displays informationby projecting an image on a wall surface and the like, the display unit110 may be configured from the device projecting that image or a displaysurface on which that device displays the projected image.

As described above, the camera 120, which is provided in the samedirection as the display surface of the display unit 110, capturesimages of objects facing the display surface of the display unit 110.The camera 120 includes, for example, an image sensor such as a CCDimage sensor or a CMOS image sensor. The camera 120 can obtain data thatwill serve as the basis of the image data by converting light that isincident on the image sensor via a lens into an electric signal. Thecamera 120 supplies the electric signal to the image processing unit130.

The image processing unit 130 obtains image data by performingpredetermined signal processing on the electric signal obtained by thecamera 120. Examples of predetermined signal processing performed by thecamera 120 include noise reduction processing, white balance processingand the like. The image processing unit 130 supplies the image dataobtained by performing signal processing to the operation detection unit140.

The operation detection unit 140 detects operations made by the userfacing the display surface of the display unit 110 using the image datasupplied from the image processing unit 130. When a user operation isdetected using the image data, the operation detection unit 140 outputsa detection result to the area selection unit 150. During detection ofuser operations using the image data, the operation detection unit 140executes, for example, face recognition processing and processing fordetecting a predetermined operation by the user.

The area selection unit 150 selects an area from among the display areaof the display unit 110 using the user operation detection result fromthe operation detection unit 140. The area selection processingperformed by the area selection unit 150 will be described in moredetail below. When an area has been selected from among the display areaof the display unit 110, the area selection unit 150 suppliesinformation about the selected area to the information display unit 160.

The information display unit 160 makes the display unit 110 displaypredetermined information on the area in the display area of the displayunit 110 selected by the area selection unit 150. The predeterminedinformation displayed by the information display unit 160 on the displayunit 110 is information that lets the user perform an operation or makea selection, such as a cursor, an icon, a menu screen or the like.

A function configuration example of the display apparatus 100 accordingto an embodiment of the present disclosure was described above withreference to FIG. 2. Next, a function configuration example of anoperation detection unit 140 included in the display apparatus 100according to an embodiment of the present disclosure will be described.

FIG. 3 is an explanatory diagram illustrating a function configurationexample of the operation detection unit 140 included in the displayapparatus 100 according to an embodiment of the present disclosure. Afunction configuration example of the operation detection unit 140included in the display apparatus 100 according to an embodiment of thepresent disclosure will now be described with reference to FIG. 3.

As illustrated in FIG. 3, the operation detection unit 140 included inthe display apparatus 100 according to an embodiment of the presentdisclosure includes the face detection unit 142 and a gesture detectionunit 144.

Using the image data supplied from the image processing unit 130, theface detection unit 142 detects whether a face is included in the imagedata. Further, if a face is included in the image data, the facedetection unit 142 estimates what position in the display area of thedisplay unit 110 is indicated by that face.

Using the image data supplied from the image processing unit 130, thegesture detection unit 144 detects a predetermined gesture performed bythe user toward the display unit 110. The operation detection unit 140outputs a face detection result by the face detection unit 142 and agesture detection result by the gesture detection unit 144 to the areaselection unit 150. The area selection unit 150 can use the facedetection result by the face detection unit 142 and the gesturedetection result by the gesture detection unit 144 supplied from theoperation detection unit 140 to select an area from among the displayarea of the display unit 110.

A function configuration example of the operation detection unit 140included in the display apparatus 100 according to an embodiment of thepresent disclosure was described above. Next, operation of the displayapparatus 100 according to an embodiment of the present disclosure willbe described.

The display apparatus 100 according to an embodiment of the presentdisclosure can detect the face of a user who is facing the displayscreen of the display unit 110 in the display apparatus 100 and gesturesmade by the user from the image data obtained by capturing an image withthe camera 120. The display apparatus 100 can select an area from thedisplay area of the display unit 110 in the display apparatus 100 (e.g.,a position or a range on the display area) based on a detection resultof the user's face and gestures made by the user. Further, the displayapparatus 100 can also display information (e.g., a cursor, a menu etc.)on that selected area.

In the following description, as an example of the operation of thedisplay apparatus 100 according to an embodiment of the presentdisclosure, processing will be described for detecting the face andgestures of a user who is facing the display surface of the display unit110 in the display apparatus 100 from image data obtained by capturingan image with the camera 120, and displaying a cursor on a positioncorresponding to the orientation of the user's face on the display unit110. Further, processing will also be described for detecting a gesturemade by a user facing the display surface of the display unit 110 in thedisplay apparatus 100, and moving the cursor based on that gesture.

Processing performed by the face detection unit 142 for estimating aface position in a display unit-centered coordinate system centered onthe display unit 110 in the display apparatus 100 that uses a facedetection position based on image coordinates will now be described indetail. First, the face detection unit 142 converts a face detectionposition based on image coordinates into an image-centered coordinatesystem face position, and then converts this face position into a faceposition in a camera-centered coordinate system. Then, the facedetection unit 142 converts the face position in a camera-centeredcoordinate system into a face position in a display unit-centeredcoordinate system centered on the display unit 110 in the displayapparatus 100.

FIG. 4 is an explanatory diagram illustrating conversion processing intoan image-centered coordinate system face position performed by the facedetection unit 142. The conversion from an image coordinate-based facedetection position (x,y) (wherein 0≦x, y≦1) into an image-centeredcoordinate system face position (v_rad,h_rad) (rad) is determinedaccording to the following formulae.

v-rad=face_cy*camViewAngleH

h-rad=face_cx*camViewAngleH

The respective variables have the following meanings.

face_cx: Horizontal direction face detection position in animage-centered coordinate system (−0.5 to 0.5)face_cy: Perpendicular direction face detection position in animage-centered coordinate system (−0.5 to 0.5)face_cx: x+w/2+0.5face_cy: x+h/2+0.5camViewAngleV: Camera perpendicular direction angle of field (rad)camViewAngleH: Camera horizontal direction angle of field (rad)w: Face detection width (0 to 1)h: Face detection height (0 to 1)

FIG. 5 is an explanatory diagram illustrating processing performed bythe face detection unit 142 to convert a face position in animage-centered coordinate system into a face position in acamera-centered coordinate system. The conversion from the face position(v_rad,h_rad) (rad) in an image-centered coordinate system into the faceposition (cam_x,cam_y,cam_z) (mm) in a camera-centered coordinate systemcan be performed according to formulae like the following.

cam_x=1000*FACESIZE_AT_1000/w

cam_y=cam_x*tan(h_rad)

cam_z=cam_x*tan(v_rad)

Note that FACESIZE_AT_1000 represents the face size w (0 to 1) when at adistance of 1000 (mm).

FIG. 6 is an explanatory diagram illustrating processing performed bythe face detection unit 142 to convert a face position in acamera-centered coordinate system into a face position in a displayunit-centered coordinate system centered on the display unit 110 in thedisplay apparatus 100. The conversion from the face position(cam_x,cam_y,cam_z) (mm) in a camera-centered coordinate system into theface position (pos_x, pos_y, pos_z) (mm) in a display unit-centeredcoordinate system centered on the display unit 110 in the displayapparatus 100 can be performed according to formulae like the following.

pos_x=cam_x*cos(pitch)+cam_z*sin(pitch)+shift_x

pos_y=cam_y+shift_y

pos_z=cam_x*sin(pitch)+cam_z*cos(pitch)+shift_z

The respective variables have the following meanings.

pitch: Slope (rad) with respect to the display apparatus 100shift_{x,y,z}: Deviation (mm) in the x, y, or z direction from thecenter of the camera 120 to the center of the display unit 110 in thedisplay apparatus 100

Thus, the face detection unit 142 can determine the face position in adisplay unit-centered coordinate system from a face detection positionbased on image coordinates.

The face detection unit 142 acquires the face detection position (x,y)(0 to 1) and the face orientation (theta,phi) (rad) that the face isfacing. Based on a calculation, the face detection unit 142 candetermine a position on a display area of the display unit 110 indicatedby that face from the face position and the face orientation in adisplay unit-centered coordinate system. FIG. 7 is an explanatorydiagram illustrating calculation of a position on a display area of adisplay unit 110 indicated by a face detected by the face detection unit142.

From the face position (pos_x, pos_y, pos_z) (mm) in a displayunit-centered coordinate system and the face orientation, the conversioninto a position (dsp_x, dsp_y, dsp_z) (mm) on the display area of thedisplay unit 110 indicated by that face can be performed according tothe formulae like the following.

theta=a tan(pas_y/pos_x)

phi=a tan(pas_z/pos_x)

The respective variables have the following meanings.

dsp_x=0.0

dsp_y=−pos_y+pos_x*tan(yaw+theta)

dsp_z=−pos_z+pos_x*tan(pitch+phi)

FIG. 8 is an explanatory diagram illustrating processing to convert aposition on a screen indicated by a face detected by the face detectionunit 142 into a cursor position in an image coordinate system. Theconversion of the position (dsp_x, dsp_y, dsp_z) (mm) on a screenindicated by a face detected by the face detection unit 142 into acursor position (csr_x, csr_y) (pixel) in an image coordinate system canbe performed according to formulae like the following.

csr_x=(−dsp_y/dsp_size_w+0.5)*dsp_rsl_w

csr_y=(−dsp_z/dsp_size_h+0.5)*dsp_rsl_h

The respective variables have the following meanings.

dsp_size_w: Display horizontal width length (mm)dsp_size_h: Display vertical width length (mm)dsp_rsl_w: Display horizontal width resolution (pix)dsp_rsl_h: Display vertical width resolution (pix)

Thus, from the face position and the face orientation in acamera-centered coordinate system, the area (coordinates) on the displayarea of the display unit 110 indicated by that face can be determined.

Next, the gesture detection processing performed by the gesturedetection unit 144 will be described in more detail based on a specificexample. For example, when the user faces the camera 120, and changeshis/her hand into a specific shape, such as a fist (“rock” inrock-paper-scissors), or an open palm (“paper” in rock-paper-scissors),the hand is detected by the gesture detection unit 144, and tracking ofthe hand movements (hand tracking) is started.

When the fact that a specific operation was performed by the user isdetected from the image data, the gesture detection unit 144 estimatesthe hand position in a camera-centered coordinate system from thedetected position based on the image coordinates. The estimationprocessing of the hand position by the gesture detection unit 144 iscarried out in a similar manner to the above-described face detectionunit 142.

The gesture detection unit 144 determines a cursor position(csr_x_(t+dt), csr_y_(t+dt)) in an image coordinate system after apredetermined time dt according to the following formulae based on themovement amount (dHw, dHh) (mm/s) in the horizontal and perpendiculardirections for the estimated hand position in a camera-centeredcoordinate system.

csr_x _(t+dt)=csr_x _(t)+gain*dHw

csr_y _(t+dt)=csr_h _(t)+gain*dHh

Note that “gain” is a coefficient. By varying the gain value, themovement amount of the cursor based on hand tracking can be adjusted.

The only time that the face information has an influence on the cursoris when the hand tracking starts. While the hand tracking is continuing,the face information does not have any influence on the movement of thecursor. While the hand tracking is continuing, the display apparatus 100controls the cursor using information about the position of the user'shand that has been detected by the gesture detection unit 144.

The display apparatus 100 according to an embodiment of the presentdisclosure can easily move the cursor to a target position by performingthe above-described operations so that large movements of the cursor aremade using face information, and after the cursor has been moved usingface information, finer movements of the cursor are made based onmovement of the hand.

FIGS. 9A to 9C are explanatory diagrams illustrating control examples ofthe display apparatus 100 according to an embodiment of the presentdisclosure. FIGS. 9A to 9C illustrate examples of controlling themovement of the cursor displayed on the display unit 110 based on theuser facing the display apparatus 100 according to an embodiment of thepresent disclosure and moving his/her face and/or performing apredetermined gesture. In FIGS. 9A to 9C, examples are illustrated of aweb browser displayed by the display apparatus 100, which is operatedbased on the user facing the display apparatus 100 and moving his/herface and/or raising his/her arm.

FIG. 9A illustrates a state in which, with a cursor 111 being displayedby the information display unit 160 on the display unit 110 in thedisplay apparatus 100, a user facing the display apparatus 100 looks inthe direction of a given location L1 of the display unit 110 that is nota display position of the cursor 111. Thus, when the user is simplylooking in the direction of the given location L1 of the display unit110, the information display unit 160 does not move the display positionof the cursor 111 to the location L1.

FIG. 9B illustrates a state in which, with the cursor 111 beingdisplayed by the information display unit 160 on the display unit 110 inthe display apparatus 100, a user facing the display apparatus 100 looksin the direction of the given location L1 of the display unit 110 thatis not a display position of the cursor 111, and performs a gesture inwhich his/her hand is raised. When the user performs a gesture ofraising his/her hand while looking in the direction of the givenlocation L1 of the display unit 110, the camera 120 captures an image ofthe user, the operation detection unit 140 detects the orientation ofthe user's face and the user gesture from the captured image data, thearea selection unit 150 selects the position (location L1) of thedisplay unit 110 based on the detection result of the operationdetection unit 140, and the information display unit 160 displays thecursor at the position selected by the area selection unit 150.

Note that when changing from the state in FIG. 9A to the state in FIG.9B, the display apparatus 100 can move the cursor 111 in a manner thatlets the user see the movement trajectory of the cursor 111, or caninstantly move the cursor 111 without showing the trajectory. Further,when changing from the state in FIG. 9A to the state in FIG. 9B, thedisplay apparatus 100 may highlight the cursor 111 (e.g., thesurrounding of the cursor 111 may be lit up) to let the user know thatthe cursor 111 has moved.

FIG. 9C illustrates a state in which, following on from the stateillustrated in FIG. 9B, a user facing the display apparatus 100 moveshis/her raised right hand. When a user facing the display apparatus 100moves his/her raised right hand following on from the state illustratedin FIG. 9B, the camera 120 captures an image of the user, the operationdetection unit 140 detects the user gesture from the captured imagedata, the area selection unit 150 selects the position of the displayunit 110 based on the detection result of the operation detection unit140, and the information display unit 160 displays the cursor at theposition selected by the area selection unit 150. Namely, if the usermoves his/her hand to the right, the information display unit 160performs a control to move the cursor to the right, and if the usermoves his/her hand to the left, the information display unit 160performs a control to move the cursor to the left.

As illustrated in FIG. 9C, when the display apparatus 100 moves thecursor 111 by tracking the movement of the user's arm, the displayapparatus 100 may select a link on a web page displayed by a webbrowser, for example, when it has been detected that the user hasperformed a separate operation (e.g., lowering a raised arm in front ofthe body etc.).

Note that, as illustrated in FIG. 9C, even if a user facing the displayapparatus 100 changes the orientation of his/her face while movinghis/her raised right hand, the display apparatus 100 does not track thechanges in the orientation of his/her face when displaying the cursor.To make large movements of the cursor position, for example, the userlowers his/her right arm, shifts the orientation of his/her face tomatch the position that he/she wants to the cursor to be displayed,raises his/her right arm once more, and clenches and opens his/her hand.When the display apparatus 100 detects that the user has lowered his/herright arm and then raised his/her right arm once more, the displayapparatus 100 executes a control to display the cursor at the positionthat the user's face is looking at that point.

Thus, the display apparatus 100 can control movement of the cursordisplayed on the display unit 110 based on a result obtained bycapturing an image of the user with the camera 120.

FIG. 10 is a flow diagram illustrating operation of the displayapparatus 100 according to an embodiment of the present disclosure. Theflow diagram of FIG. 10 illustrates the operations performed when thedisplay apparatus 100 recognizes a user's face and a gesture from imagedata captured by the camera 120, and displays information on the displayunit 110 based on the recognition result. The operations performed bythe display apparatus 100 according to an embodiment of the presentdisclosure will now be described with reference to FIG. 10.

To recognize a user's face and a gesture, and display information on thedisplay unit 110 based on the recognition result, first, the displayapparatus 100 captures an image of the user's outline with the camera120 included in the display apparatus 100 (step S101). Then, when animage of the user's shape has been captured with the camera 120 includedin the display apparatus 100, the display apparatus 100 detects theorientation of the user's face and a user gesture from the image datacaptured by the camera 120 (step S102). The detection of the orientationof the user's face and the user gesture is carried out by the operationdetection unit 140. The detection of the orientation of the user's faceand the user gesture by the operation detection unit 140 is performedbased on processing such as that described above.

Then, when the orientation of the user's face and the user gesture hasbeen detected from the image data captured by the camera 120 in stepS102, the display apparatus 100 selects an area on the display area ofthe display unit 110 based on the detected orientation of the user'sface and user gesture (step S103). The selection of the area on thedisplay area of the display unit 110 based on the detected orientationof the user's face and user gesture is executed by the area selectionunit 150.

Then, when the area on the display area of the display unit 110 has beenselected based on the detected orientation of the user's face and usergesture in step S103, the display apparatus 100 displays information (anicon, a menu etc.) on the area on the display area of the display unit110 selected in step S103 (step S104). The display of information on thearea on the display area of the display unit 110 selected in step S103is executed by the information display unit 160.

Then, when the display apparatus 100 has displayed in step S104 theinformation (an icon, a menu etc.) on the area on the display area ofthe display unit 110 selected in step S103, the display apparatus 100detects the user gesture performed subsequently after the gesturedetected in step S102, and changes the selected area on the display areaof the display unit 110 based on that gesture (step S105). The changingof the selection area on the display area of the display unit 110 isexecuted by the area selection unit 150.

The processing performed in step S105 corresponds to the controlillustrated in FIG. 9C. Namely, after the user's right arm has beenraised, if the operation detection unit 140 detects that the right armhas been moved, the area selection unit 150 changes the selection areabased on the detection result from the operation detection unit 140, andthe information display unit 160 moves the cursor 111 displayed on thedisplay unit 110 based on the selection area changed by the areaselection unit 150.

By executing such a series of processes, the display apparatus 100 candetect the orientation of the user's face and user gestures, select anarea from among the display area of the display unit 110 using thatdetection result, and display information on that selected area.

For example, if the user is looking in the direction of the display unit110, when the display apparatus 100 detects the position that the useris looking from among the display area of the display unit 110, andfurther detects that the user has performed a specific operation (e.g.,raising of the right arm), the display apparatus 100 displays the cursorat the position that the user is looking. Then, in a state in which thecursor is displayed on the display unit 110 based on the user havinglooked in the direction of the display unit 110 and performed theabove-described specific operation, if the display apparatus 100 detectsthat the user has performed another separate specific operation (e.g.,moving the raised right arm), the display apparatus 100 can move thecursor displayed on the display unit 110 based on that operation.

Operation of the display apparatus 100 was described above. In the abovedescription, although an example was described in which the displayapparatus 100 displayed a web browser, and that web browser was operatedusing a cursor, obviously the present disclosure is not limited to suchan example.

For example, when the display apparatus 100 is displaying content suchas moving images or a still image, even if the user is made to perform acontrol such as selecting or playing/stopping the content, the displayapparatus 100 can execute the above-described control based on detectionof the face orientation and/or a gesture.

FIG. 11 is an explanatory diagram illustrating an example of a screendisplayed on the display unit 110 in the display apparatus 100. FIG. 11illustrates a state in which, when content C1 is displayed on thedisplay unit 110, a menu M1 is displayed based on a user gesture. Todisplay this menu M1, for example, the user looks at the area on whichthe menu M1 is displayed, and performs a specific operation (e.g.,raising his/her right arm). The display apparatus 100 detects theorientation of the user's face when the user performed the specificoperation. If the user's face is looking at the area on which the menuM1 is displayed, the menu M1 is displayed on the display unit 110 by theinformation display unit 160.

Note that although an example is illustrated in FIG. 11 in which themenu M1 is displayed on an area on the right side of the display unit110, the present disclosure is not limited to such an example. Forexample, when the operation detection unit 140 detects that theabove-described specific operation has been performed with the userlooking at an area on the left side of the display unit 110, the menu M1may be displayed on an area on the left side of the display unit 110.Further, for example, if a separate menu different to menu M1 is to bedisplayed on an area on the left side of the display unit 110, thedisplay apparatus 100 may be configured so that the menu M1 is displayedwhen the user looks at an area on the right side of the display unit 110and performs a specific operation, and a menu different to the menu M1is displayed when the user looks at an area on the left side of thedisplay unit 110 and performs a specific operation.

In FIG. 11, a case is illustrated in which, when the display apparatus100 is displaying content such as moving images or a still image, theuser is made to perform a control such as selecting or playing/stoppingthe content. Next, a case will be described in which the user is made toselect one area from among a plurality of areas of the display unit 110.

FIG. 12 is an explanatory diagram illustrating an example of a screendisplayed on the display unit 110 in the display apparatus 100. In FIG.12, an example is illustrated in which the user is made to select onearea from among 6 areas R1 to R6. Note that what is displayed in the sixareas R1 to R6 may be content such as moving images or a still image, ormay be a reduced-size display of a web page displayed by the webbrowser.

To select one area from among the six areas R1 to R6, the user looks inthe direction of one of the six areas R1 to R6, and performs a specificoperation (e.g., raises his/her right arm). The display apparatus 100detects the orientation of the user's face when the user performed thespecific operation. If the user is looking at one of the areas R1 to R6,the display apparatus 100 selects that area and then, for example,enlarges the reduced-size web page and displays it across the wholescreen.

The display apparatus 100 may also detect the orientation of the user'sface when the user performed the specific operation, and change thedisplayed operation based on the direction that the face is looking. Forexample, if the user wants to vertically scroll up or down avertically-long web page displayed on the display unit 110 by thedisplay apparatus 100, the user performs a specific operation (e.g.,raises his/her right arm) while looking at an upper section or a lowersection of the display area of the display unit 110. If the user waslooking at an upper section of the display area of the display unit 110when that operation was performed, the display apparatus 100 can scrollthe web page displayed on the display unit 110 upwards, and if the userwas looking at a lower section of the display area of the display unit110, the display apparatus 100 scrolls the web page displayed on thedisplay unit 110 downwards. Even when the user wants to scroll in aleft-right direction, scrolling in the left-right direction can beperformed by similarly detecting the orientation of the user's face.

When the user operates the cursor by moving his/her arm, there is alimit to the range that the user can move his/her arm. Therefore, whenthe user has performed a specific operation (e.g., raising his/her rightarm), the display apparatus 100 may display on the display unit 110 themovable range over which the user can operate.

FIGS. 13A and 13B are explanatory diagrams illustrating examples of ascreen displayed on the display unit 110 in the display apparatus 100.FIGS. 13A and 13B illustrate examples in which a movable range 51 overwhich the user can operate the cursor 111 is displayed on the displayunit 110 by the display apparatus 100 when the user has performed aspecific operation (e.g., raising his/her right arm). Note that theuser's movable range may be determined from the orientation of theuser's face and the user's posture when the user performed the specificoperation.

For example, when the user displays the cursor 111 at a location wherethe user is looking by raising his/her right arm, if the right arm israised near his/her face, the movable range 51 of the cursor 111 is, asillustrated in FIG. 13A, defined by a roughly circular shape centered onthe position where the cursor 111 is displayed. On the other hand, ifthe right arm is raised far away from his/her face, the movable range Siof the cursor 111 is a shape like that illustrated in FIG. 13B.

The display apparatus 100 may change the operation control based ondifferences with a detected specific operation. For example, the displayapparatus 100 may be configured so as to, when the user has raisedhis/her right arm, execute a different operation by recognizingdifferences in the shape of the hand between when the user has againclenched his/her right hand or whether the right hand is open.

The display apparatus 100 can prompt the user to perform a resetoperation for resetting the processing if the position of the processingtarget and the position of the detected face are separated by apredetermined amount or more. For example, if a specific operation(e.g., raising his/her right arm) by the user is detected and the cursoris displayed on the display unit 110, if the position that the user'sface is looking at while the user is facing the display unit 110 and theposition of the cursor displayed on the display unit 110 are separatedby a predetermined distance, the display apparatus 100 can display onthe display unit 110 information prompting the user to perform a resetoperation for resetting the cursor display.

The display apparatus 100 can also change the speed gain of the cursordisplayed on the display unit 110 based on, after the user performs aspecific operation and the cursor is displayed on the display unit 110,a distance between the position of the user's face and the position ofthe site where the user performed the specific operation. For example,if the distance is close between the position of the user's face and theposition of the site where the user performed the specific operation,the movement speed of the cursor based on movement of the user's arm maybe decreased, while if the distance is far between the position of theuser's face and the position of the site where the user performed thespecific operation, the movement speed of the cursor based on movementof the user's arm may be increased.

After the cursor is displayed on the display unit 110, the displayapparatus 100 can also change the movement speed of the cursor based onthe presence/absence of content that can be selected around the cursor.For example, after the cursor is displayed on the display unit 110, ifthere is a plurality of content that can be selected around the cursor,in order to facilitate content selection, the display apparatus 100 candecrease the movement speed of the cursor, and if there is only onepiece of content or no content at all that can be selected around thecursor, to emphasize cursor operability, the display apparatus 100 canincrease the movement speed of the cursor.

Although in the above description the display apparatus 100 displayedthe cursor on the display unit 110 based on the orientation of the facewhen the user performed a specific operation, especially a physicaloperation, the present disclosure is not limited to this. For example,the display apparatus 100 can display the cursor on the display unit 110based on the orientation of the user's face when the user issued aspecific sound.

Next, a hardware configuration of the display apparatus 100 according toan embodiment of the present disclosure will be described in more detailwith reference to FIG. 14. FIG. 14 is a block diagram illustrating ahardware configuration of the display apparatus 100 according to anembodiment of the present disclosure.

The display apparatus 100 mainly includes a CPU 901, a ROM 903, and aRAM 905. Further, the display apparatus 100 further includes a host bus907, a bridge 909, an external bus 911, an interface 913, an inputdevice 915, an output device 917, a storage device 917, a drive 921, aconnection port 923, and a communication apparatus 925.

The CPU 901, which functions as a calculation processing device and acontrol device, controls all or a part of the operations in the displayapparatus 100 based on various programs recorded in the ROM 903, RAM905, storage device 917, or a removable recording medium 927. The ROM903 stores programs, calculation parameters and the like used by the CPU901. The RAM 905 temporarily stores the programs to be used by the CPU901, and parameters that appropriately change during program execution.These units are connected to each other by the host bus 907, which isconfigured from an internal bus such as a CPU bus.

The host bus 907 is connected to the external bus 911, which is a PCI(peripheral component interconnect/interface) bus or the like.

The input device 915 is an operation device that is operated by theuser, such as a mouse, keyboard, touch panel, button, switch, lever andthe like. The input device 915 may be, for example, a remote controldevice (a so-called “remote control”) that utilizes infrared rays orother radio waves, or may be an external connection device 929, such asa mobile telephone or a PDA that supports the operations of the displayapparatus 100. The input device 915 includes, for example, an inputcontrol circuit that generates an input signal based on informationinput by the user using the above-described operation device, andoutputs the generated input signal to the CPU 901. The user of thedisplay apparatus 100 can input various types of data into the displayapparatus 100 and issue processing operation instructions by operatingthis input device 915.

The output device 917 is configured from a device that can visually oraurally notify the user of acquired information. Examples of such adevice include a display device such as a CRT display, a liquid crystaldisplay device, a plasma display panel device, an organic EL display,and a lamp, an audio output device such as a speaker or headphones, aprinter apparatus, a mobile telephone, a facsimile and the like. Theoutput device 917 outputs results obtained based on various processesperformed by the display apparatus 100, for example. Specifically, thedisplay apparatus displays results obtained based on various processesperformed by the display apparatus 100 as text or an image. On the otherhand, the output device 917 converts played-back audio data or an audiosignal formed from audio data into an analog signal, and outputs theconverted analog signal.

The storage device 919 is a device for storing data that is configuredas an example of the storage unit of the display apparatus 100. Thestorage device 919 is configured from, for example, a magnetic storageunit device such as a HDD (hard disk drive), a semiconductor storagedevice, an optical storage device, a magneto-optical storage device andthe like. This storage device 919 stores programs and various types ofdata executed by the CPU 901, and various types of externally-acquireddata, for example.

The drive 921 is a reader/writer for a recording medium, and isinstalled in the display apparatus 100 or is externally attached. Thedrive 921 reads information recorded on a mounted removable recordingmedium 927, such as a magnetic disk, an optical disc, a magneto-opticaldisk, or a semiconductor memory, and outputs the read information to theRAM 905. Further, the drive 921 can also write information onto amounted removable recording medium 927, such as a magnetic disk, anoptical disc, a magneto-optical disk, or a semiconductor memory, andoutputs the read information to the RAM 905. Examples of the recordingmedium 927 include CompactFlash® (CF), a flash memory, or a SD memorycard (secure digital memory card) and the like. Further, the recordingmedium 927 may also be a IC card (integrated circuit card) on which anon-contact IC chip is mounted, an electronic device and the like.

The connection port 923 is a port for directly connecting a device tothe display apparatus 100. Examples of the connection port 923 include,for example, a USB (universal serial bus) port, an IEEE 1394 port, aSCSI (small computer system interface) port and the like. Furtherexamples of the connection port 923 include an RS-232C port, an opticalaudio terminal, a HDMI (high-definition multimedia interface) port andthe like. By connecting the external connection device 929 to theconnection port 923, the display apparatus 100 directly acquires varioustypes of data from the external connection device 929 and providesvarious types of data to the external connection device 929.

The communication apparatus 925 is a communication interface configuredfrom a communication device for connecting to a communication network931, for example. The communication apparatus 925 may be a wired or awireless LAN (local area network), Bluetooth®, or WUSB (wireless USB)communication card, for example. Further, the communication apparatus925 may be an optical communication router, an ADSL (asymmetric digitalsubscriber line) router, or a modem used for various types ofcommunication. This communication apparatus 925 can, for example,transmit and receive signals and the like based on a predeterminedprotocol such as TCP/IP, for example, to/from the Internet or anothercommunication device. In addition, the communication network 931connected to the communication apparatus 925 is configured from a wiredor wirelessly connected network, and may be, for example, the Internet,a home LAN, infrared communication, radio wave communication, satellitecommunication or the like.

In the above, an example was illustrated of hardware configuration thatcan be executed by the display apparatus 100 according to an embodimentof the present disclosure. The above-described constituent elements maybe configured using multi-purpose parts or from hardware specialized forthe function of each constituent element. Therefore, the utilizedhardware configuration may be appropriately modified based on thetechnological level at the time of implementing the present disclosure.

2. Summary

According to the display apparatus 100 according to an embodiment of thepresent disclosure as described above, the position of the face of auser who is looking at a display unit 110 in the display apparatus 100and a user operation are detected. When the display apparatus 100according to an embodiment of the present disclosure detects that theuser has performed a specific operation, the display apparatus 100selects an area of the display unit 110 based on the position of theface at that time, and displays predetermined information (a cursor or amenu) on that selected area.

The display apparatus 100 according to an embodiment of the presentdisclosure can facilitate a remote operation that is performed based ondetection of a specific gesture when the user tries to perform agesture-based remote operation.

Operation of the display apparatus 100 according to an embodiment of thepresent disclosure described above can be performed by hardware or bysoftware. If performing by software, for example, operation can beperformed by a CPU or some other control apparatus provided in thedisplay apparatus 100 reading and sequentially executing computerprograms stored on a recording medium, such as a ROM, provided in thedisplay apparatus 100.

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

Additionally, the present technology may also be configured as below.

(1) An information processing apparatus including:

an operation detection unit configured to detect an orientation of auser's face and operations performed by the user; and

an area selection unit configured to, when the operation detection unitdetects that the user has performed a first operation, select an area ona screen based on the orientation of the user's face during the firstoperation.

(2) The information processing apparatus according to (1), wherein whenthe operation detection unit detects that the user has switched from thefirst operation to a second operation, the area selection unit isconfigured to change the selected area based on the second operation.(3) The information processing apparatus according to (1) or (2),further including

an information display unit configured to display information on thescreen,

wherein the information display unit is configured to displayinformation on an area on the screen selected by the area selectionunit.

(4) The information processing apparatus according to (3), wherein theinformation display unit is configured to display information based on aposition of the user's face detected by the operation detection unit.(5) The information processing apparatus according to (3) or (4),wherein the information display unit is configured to change thedisplayed information based on a content of the first operation detectedby the operation detection unit.(6) The information processing apparatus according to any one of (3) to(5), wherein the information display unit is configured to change adisplay when the user has performed the first operation based on theorientation of the face detected by the operation detection unit.(7) The information processing apparatus according to any one of (3) to(6), wherein the information display unit is configured to display acursor on the area on the screen selected by the area selection unit.(8) The information processing apparatus according to any one of (3) to(7), wherein the information display unit is configured to display onthe screen a user operable area at a point when the user performed thefirst operation.(9) The information processing apparatus according to any one of (3) to(8), wherein, if the area selected by the area selection unit and thearea selected by the area selection unit based on the orientation of theuser's face detected by the operation detection unit are separated by apredetermined amount or more, the information display unit is configuredto display on the screen information prompting a reset operation forresetting selection by the area selection unit.(10) The information processing apparatus according to any one of (2) to(9), wherein when the operation detection unit detects that the user hasswitched from the first operation to the second operation, the areaselection unit is configured to change a speed for changing the areaselected based on the second operation, based on a distance between aposition where the user performed the second operation and a position ofthe user's face detected by the operation detection unit.(11) The information processing apparatus according to any one of (2) to(10), wherein when the operation detection unit detects that the userhas switched from the first operation to the second operation, the areaselection unit is configured to change the area selected based on thesecond operation, and change a speed for changing the area selectedbased on the second operation based on a number of pieces of contentaround the selected area.(12) An information processing method including:

detecting an orientation of a user's face and operations performed bythe user; and

when performance of a first operation by the user is detected, selectingan area on a screen based on the orientation of the user's face duringthe first operation.

(13) A computer program configured to make a computer:

detect an orientation of a user's face and operations performed by theuser; and

when performance of a first operation by the user is detected, select anarea on a screen based on the orientation of the user's face during thefirst operation.

What is claimed is:
 1. An information processing apparatus comprising atleast one processor configured to: obtain orientation information of aface of a user; obtain first gesture information of the user, the firstgesture information being related to at least one of a firsthand-gesture and a first arm-gesture made by the user; select a partialarea contained in a screen in accordance with the orientationinformation and the first gesture information; and cause a first imagerelated to the selected partial area to be changed in accordance withthe first gesture information.
 2. The information processing apparatusaccording to claim 1, wherein the first image is an image which the useroperates or selects.
 3. The information processing apparatus accordingto claim 2, wherein the at least one processor is further configured tomove the first image to the selected partial area in accordance with thefirst gesture information.
 4. The information processing apparatusaccording to claim 3, wherein when moving the first image to theselected partial area, the at least one processor control display of asecond image in accordance with the movement of the first image.
 5. Theinformation processing apparatus according to claim 2, wherein the atleast one processor is further configured to: obtain second gestureinformation, the second gesture information being related to at leastone of a second hand-gesture and a second arm-gesture; and controldisplay of a third image in accordance with the second gestureinformation while the first image is selected or operated upon.
 6. Theinformation processing apparatus according to claim 5, wherein the thirdimage is an image related to a link on a web page displayed by a webbrowser, and the at least one processor is further configured to selectthe link on the web page in accordance with the second gestureinformation.
 7. The information processing apparatus according to claim2, wherein after selecting the partial area, the at least one processoris further configured to move the first image in accordance with thefirst gesture information without using the orientation information. 8.The information processing apparatus according to claim 7, whereinduring a time the at least one processor does not obtain the firstgesture information, the least one processor does not move the firstimage regardless of the obtained orientation information.
 9. Theinformation processing apparatus according to claim 2, wherein the firstimage is at least one of a cursor, an icon and a menu.
 10. Theinformation processing apparatus according to claim 9, wherein the firstimage is the menu including a first menu and a second menu, the partialarea includes a first partial area and a second partial area, and the atleast one processor is further configured to control display of thefirst menu or the second menu in accordance with the first partial areaor the second partial area which the face of the user is oriented to.11. The information processing apparatus according to claim 1, whereinthe at least one processor is further configured to enlarge the firstimage in accordance with the first gesture information.
 12. Theinformation processing apparatus according to claim 1, wherein the firstimage is a webpage, and the at least one processor is further configuredto scroll, in accordance with the first gesture information, the webpagein a direction based on the orientation information.
 13. The informationprocessing apparatus according to claim 1, wherein when causing thefirst image to be changed in accordance with the first gestureinformation, the at least one processor controls display of a fourthimage having a substantially circular shape.
 14. The informationprocessing apparatus according to claim 1, wherein the at least oneprocessor is further configured to control display of the first image inaccordance with a distance between the face and a site of the user, thesite of the user corresponding to the first gesture information.
 15. Theinformation processing apparatus according to claim 1, wherein the firstimage is a content image that is a moving image or a still image. 16.The information processing apparatus according to claim 15, wherein theat least one processor is further configured to control display of thecontent image in accordance with the orientation information.
 17. Theinformation processing apparatus according to claim 1, wherein the atleast one processor is further configured to obtain the orientationinformation from an imaging device connected to the screen.
 18. Theinformation processing apparatus according to claim 1, wherein the atleast one processor is further configured to obtain the orientationinformation from an imaging device apart from the user.
 19. Aninformation processing method comprising: obtaining orientationinformation of a face of a user; obtaining gesture information of theuser, the gesture information being related to at least one of ahand-gesture and an arm-gesture made by the user; selecting a partialarea contained in a screen in accordance with the orientationinformation and the gesture information; and causing an image related tothe selected partial area to be changed in accordance with the gestureinformation.
 20. A non-transitory computer-readable medium havingembodied thereon a program, which when executed by a computer causes thecomputer to execute a method, the method comprising: obtainingorientation information of a face of a user; obtaining gestureinformation of the user, the gesture information being related to atleast one of a hand-gesture and an arm-gesture made by the user;selecting a partial area contained in a screen in accordance with theorientation information and the gesture information; and causing animage related to the selected partial area to be changed in accordancewith the gesture information.